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Reliability in terms of functional properties from the safety-liveness spectrum is an indispensable 
requirement of low-level operating-system (OS) code. However, with evermore complex and thus 
less predictable hardware, quantitative and probabilistic guarantees become more and more impor- 
tant. Probabilistic model checking is one technique to automatically obtain these guarantees. First 
experiences with the automated quantitative analysis of low-level operating-system code confirm the 
expectation that the naive probabilistic model checking approach rapidly reaches its limits when in- 
creasing the numbers of processes. This paper reports on our work-in-progress to tackle the state 
explosion problem for low-level OS-code caused by the exponential blow-up of the model size when 
the number of processes grows. We studied the symmetry reduction approach and carried out our ex- 
periments with a simple test-and-test-and-set lock case study as a representative example for a wide 
range of protocols with natural inter-process dependencies and long-run properties. We quickly see 
a state-space explosion for scenarios where inter-process dependencies are insignificant. However, 
once inter-process dependencies dominate the picture models with hundred and more processes can 
be constructed and analysed. 



1 Introduction 

Safety-critical software in space, flight, and automotive control systems need not only produce correct 
results. It must produce these results in time and, to the extent possible, despite component failures. 
Worst-case execution-time (WCET) analyses (cf., (21 [161 [23j) are able to produce the former kind of 
guarantees whereas reliability techniques (cf., 11221 141171) are designed to rule out negative effects from the 
latter. However, WCET analyses only produce guarantees in the form of upper bounds on the execution 
times of all involved components, which hold even in the most extreme situations. Yet, most computer 
systems are either not safety critical or they include fail safe mechanisms that prevent damage in highly 
exceptional situations. For such systems, it is much more appropriate to look at the 99.9% quantile of 
the execution time (i.e., the execution time that is not exceeded with a probability of 99.9%), ignoring 
exceptional cases, that are dealt with by other means anyway. 

Many software systems, especially operating systems, contain optimisations that are geared towards 
the average case, incurring additional costs under exceptional circumstances. To justify such optimisa- 
tions one has to look at the probability at which the optimisation is advantageous as well as the additional 
costs that apply with a (hopefully) low probability. Here it is again often more appropriate to investigate 
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a suitable quantile of the additional costs, because pathological cases are either not interesting or solved 
by other means. 

In previous work Q, we presented a combined simulation and model checking based approach to 
determine the probabilistic quantitative properties similar to what we described in the two preceding 
paragraphs. As a part of this first case study we have extended the model checker PRISM iMTl with an 
operator for steady-state dependent properties and performed an analysis of the long run behaviour of a 
simple test-and-test-and-set spinlock [1]. However, scaling the number of processes competing for the 
lock proved difficult, as the state space to be considered by the model checker rapidly increases. 

In this paper, we report on our experience in applying symmetry reduction (6j[l0l[T3l[TTl to mitigate 
the state space explosion and thus scale our spinlock analysis. As our model is highly symmetrical, 
symmetry reduction techniques allow the analysis to be carried out on a potentially much smaller quotient 
model instead of the full model. To evaluate the benefit of applying symmetry reduction to our model and 
to gain further insight about its behaviour for an increasing number of processes, we have implemented 
a specialised tool that explores the reachable state space of the symmetry reduced quotient model and 
generates the transition matrix as input for a probabilistic model checker to carry out its analysis. This 
approach yielded a drastic improvement in the scalability of our analysis, in some cases allowing an 
analysis of models with 10,000 processes and more in contrast to the 4 to 5 processes achievable with 
the non-reduced model. 

Outline. In the next section, we introduce our general approach, present our model and demonstrate the 
scalability problems of the non-reduced model. Then, in Section [3] we discuss the symmetry reduction 
techniques that we have applied. We interpret the drastic improvements in scalability and the results of 
the analysis for an increasing number of processes. Section|4]concludes. 

2 The QuaOS Approach for Probabilistic Quantitative Properties 

In this section, we first give a short overview of the general approach that we use in the QuaOS project 
for determining probabilistic quantitative properties of operating-system code. Then, we introduce our 
cache-aware model of the test-and-test-and-set spinlock. 

One very difficult task in modelling the quantitative behaviour of low-level operating-system code 
is to design the necessary abstractions such that the model agrees in its behaviour with reality. Our 
approach is to compare the model checking results with measurements from specifically designed test 
cases. In case there are deviations, we investigate the reasons and adjust the model and, if necessary, 
the test cases to extract further details. For (H we found that a specific effect of cache coherence has a 
substantial influence on the timing of spinlocks. 

Our ultimate aim is to make predictions for hardware configurations that do not yet exist. With our 
approach, we could for instance investigate which synchronisation primitives are suitable for systems 
with more than 100 cores, depending on the work load and the performance of certain memory opera- 
tions. 

A probabilistic cache-aware model of test-and-test-and-set spinlocks. Figure Q] shows the source 
code for a test-and-test-and-set lock [1]. In the following, we refer to this lock simply as spinlock. The 
atomic_swap in line 3 writes the second argument into the location of the first argument and returns 
the old value of the first argument. Therefore, the lock has been acquired, if atomic_swap returns false. 
Otherwise, the caller spins in line 4 until the lock becomes free before it executes the more expensive 
atomic_swap again. Note that each access of the shared variable occupied might cause the transfer of the 
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respective cache line into the core-local cache. 

In our investigation we consider n identical 
parallel processes P, that use the spinlock to syn- 
chronise their critical sections. Each process p 
performs an uncritical section and a critical sec- 
tion in an endless loop. The durations of the un- 
critical and the critical sections are determined 
by probabilistic distributions, which are param- 
eters in our model. 

In (H we developed a discrete time Markov 



1 volatile bool occupied = false; 

2 void lock(){ 

3 while(atomic_swap(occupied,true)){ 

4 while(occupied){} 

5 }} 

6 void unlock(){ 

7 occupied = false 

8 } 

Figure 1 : Pseudo Code for a TTS spinlock 



chain (DTMC) model that results from the syn- 
chronous parallel composition of n processes P, (Figure [2]) and the spinlock itself (Figure [3]). Having a 
separate process for the spinlock facilitates the uniform probabilistic choice of the process that acquires 
the lock next, in case several processes are spinning. 



initialize: 
— random(v 



if t\ > then tick: 
U :=t,-\ 



if U > then tick: 
ti :=ti-\ 




if -^lockj then tick: 
tj := min{f,+l,2} 



Figure 2: Control flow graph of process p 

The control flow graph for each process p (see Figure O has the location ncrit for the non-critical 
section, wait for the time it waits for the lock, crit for the critical section and start for initialisation. In the 
locations ncrit and crit the variable ti is a timer that determines how long the process has to remain in the 
respective location. The time for the non-critical section is determined by the distribution v. The time for 
the critical section length is determined by the distributions 70 and yi ■ In the location wait the variable t { 
records whether the lock could be entered immediately (ti = 1) or whether spinning was necessary before 
obtaining the lock (ti = 2). 

We need the two distributions 70 and 71 to account for the following cache effect. In the usual 
case where the last lock owner was a different process (on a different core), the atomic_swap in Line 3 
(see Figure Q]) transfers the cache line containing the variable occupied into the local cache with state 
modified. In case the process was spinning when the lock was released, Line 4 causes an additional inter- 
core message that transfers the respective cache line into state shared. This additional inter-core message 
delays processes that were spinning before acquiring the lock by about 200 cycles. For simplicity we 
model this delay by using the distribution 71 for the critical section, which offsets 70 by this delay. For 
small critical sections (in the order of 1000 cycles) the delay significantly changes the behaviour of the 
complete system. The values for the distribution v are chosen to be at least one order of magnitude larger 
than 7o. Smaller ratios between critical and uncritical are uncommon in reality. In JH we compare our 
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Figure 3: Control flow graph of the spinlock 



results with a model that uses only one distribution for the critical section. 

To reduce state-space size, we scale down the numerical values of all distributions as far as possible 
while maintaining their ratios. The main example in this paper will be parametrised with the distributions 
7o(5) = 1, 7i(6) = 1 and v(40) = v(50) = \. 

The interaction of p with the spinlock (see Figure [3]> is a bit subtle. When P, moves into state waiti 
it signals the spinlock its desire to acquire the lock. In case the lock is free (state unlock) the spinlock 
moves in the next step to state locki, while P; stays in waiti. Having acquired the lock, p moves into state 
criti one step later. The spinlock has additional transitions between all the n lock states to permit a direct 
change from locki to lock^. 

In we have formalised four conditional long run properties, such as the probability to acquire 
the lock without waiting, the expected average waiting time, and the 95% quantile of this waiting time, 
and computed their results with the model checker PRISM. With PRISM, we were able to obtain results 
for all the considered distributions only for up to 4 processes, with 5 processes results for the simpler 
distributions were achievable. For more processes, PRISM did not terminate within several days. 

Table[T]summarises the statistics for the model with our example distributions. All calculations in the 
paper were carried out using a dual-socket Intel Xeon L5630 (Quad-Core) machine at 2.13 GHz with 192 
GB total amount of RAM. One can clearly see that increasing the number of processes in our original 
model significantly increases the complexity, resulting in a very fast blowup in the number of reachable 









Time for 


Time for calculating 


Processes 


Reachable states 


MTBDD size 


building 


steady state probabilities 


3 


67,001 


32,772 


4.05 s 


3.45 s 


4 


4,082,808 


569,042 


316.31s 


149.10 s 


5 


198,808,720 


7,632,342 


~ 20 hours 


rj 90 minutes 



Table 1: Statistics for building the model (with y {5) = 1, 71(6) = 1 and v(40) = v(50) = \) and cal- 
culating the steady state probabilities for n processes using the model checker PRISM with the sparse 
engine. The MTBDD size column lists the number of nodes used to store the symbolic transition relation 
in PRISM. 
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states as well as in the time spent analysing the model. In this paper we mainly focus on the two factors 
- building the model and calculating the steady state probabilities - that dominate the analysis time of 
the investigated conditional long run properties. We will now detail our findings in applying symmetry 
reduction as a method to scale the analysis. 

3 Applying Symmetry Reduction to the Spinlock Model 

Our model exhibits a high amount of symmetry. For example, the control flow graph for each of the 
n processes (Figure [2]) arises from a simple renaming of the processes (by their index i). Furthermore, 
the control flow graph for the spinlock (Figure [3]> treats each process identically, i.e., there is a uniform 
choice between the spinning processes to determine which particular process is awarded the lock. We 
have exploited this symmetry at a basic level already in |2J] by concentrating on a particular process (Pi) 
for the properties we analysed. Due to the symmetry between the processes, the results will be identical 
for the other processes. 

For our model, symmetry reduction is thus a natural candidate to approach the state space explosion 
problem because the n processes that compete for the spinlock behave identically. The potential of 
symmetry reduction for reducing the state space has been extensively researched in the literature in non- 
probabilistic as well as in probabilistic settings, see e.g. @[T0l[T3l|TTl[T8]|2T][8l. The basic idea is to 
perform the analysis on a smaller quotient model that arises from the identification of symmetrical states 
in such a way that the relevant behaviour for the considered properties remains unchanged. 

We will now exploit this symmetry to achieve better scalability in the number of processes for which 
the model can be effectively analysed. 

The model checker PRISM used in our previous analysis provides a built-in implementation of a 
variant of symmetry reduction |[T8l for component symmetry, i.e., where multiple components are com- 
pletely indistinguishable from each other. This approach relies on a canonicalisation of the symmetric 
states and is applied directly on the symbolic MTBDD representation of the model. Unfortunately, the 
explicit use of the process index in the spinlock module, necessary to ensure that exactly one of the 
waiting processes acquires the lock, prevented an application of this techniques. 

Symmetry reduction using generic representatives. To apply symmetry reduction, we transformed 
our model following the notion of generic representatives as described in |[TT1 [T2l |8] [9j : We pick one 
arbitrary process, P\, and treat the other n — 1 processes as indistinguishable. This allowed us to encode 
the relevant information in a more succinct form by just counting how many of the n — 1 processes 
are in each of their respective local states. As an example, let process Pj be in location ncrit2 with 
timer value t% = 10, similar to process P3 which is in location ncrit^ with timer value £3 = 10, while 
process P\ is in its location crit4 with £4 = 2. The symmetry reduced encoding just records that there 
are two processes in a local state with location ncrit and timer value t = 10 and that a single process is 
in the location crit with timer value t = 2. It is easy to see that any other permutation of the process 
indices 2,3 and 4 for this configuration would result in the same state in the symmetry-reduced quotient 
model. The transitions in the quotient model then arise by combining the effect of the transitions for a 
given configuration. For example, in a configuration where all processes are in their non-critical section 
with timer values > 0, a tick transition is enabled that corresponds to all processes staying in their non- 
critical section and decreasing their timer value by one. In the quotient model, this transition leads to the 
configuration where the number of processes in their non-critical section with timer value t corresponds 
to the number of processes in the previous configuration that are in their non-critical section with timer 
value t + 1 . The other possible combinations of state changes in the processes are handled in a similar 
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fashion. The symmetry-reduced spinlock model, again a discrete time Markov chain, then results from 
the synchronous product of P\ with the quotient model representing the processes P2 to P n and a slightly 
adapted version of the spinlock process that correctly handles the uniform distribution of the lock access 
between the waiting processes by assigning the lock to a waiting process Pi with probability w J ting and 

with 1 — w J ti to one of the other processes, where waiting is the total number of processes currently 
waiting for the lock. 

Initial distribution. In the non-reduced model, the initial length for the non-critical section of each pro- 
cess is chosen according to the distribution v, i.e., for our example distribution v(40) = v(50) = \ each 
process starts with probability | with a non-critical section of length 40 and with probability \ with a 
length of 50. In the symmetry-reduced model, the initial configuration of the symmetric processes 2 to n 
is uniformly distributed for the generic representatives instead, i.e., for v(40) = v(50) = A each assign- 
ment of x and y satisfying x + y = n — 1 is equally likely, where x represents the number of symmetric 
processes starting with length 40 and y with length 50. This modification is benign in our case, as we are 
interested in the probabilities in the long run and every state that occurs infinitely often is reached with 
probability 1 no matter the initial state. For properties that depend on the precise configuration of the 
initial states, we can straightforwardly weigh the initial generic representatives of the symmetry-reduced 
model to match the initial distribution of the original model. 

Matrix generation for MRMC. We first considered an approach similar to (U|9l, where the symmetry- 
reduced model is generated in the PRISM input language. This approach yielded some improvement, 
for example for 5 processes and v(40) = v(50) = \ as in Table [H the number of reachable states shrank 
from 198,808,720 to 8,606,543, the MTBDD size to 5,192,338, the time for building shrank from around 
20 hours to about 5.5 hours and the time for calculating the steady-state probabilities was reduced from 
around 90 minutes to about 15 minutes. On the other hand, further increasing the number of processes 
again resulted in prohibitively long times needed to build the model due to the construction of the internal 
symbolic representation of the model. In addition to using an internal calculation engine, PRISM sup- 
ports the export of the transition matrix from its internal representation of the state space. This, together 
with the exported state labels, allows the use of alternative model checkers such as the probabilistic 
model checker MRMC [ 15]. MRMC operates on a sparse representation of the transition matrix, i.e., an 
efficient, compact encoding of matrices that predominantly consist of entries with the value zero. The 
use of MRMC provides a speed-up for calculating the steady-state probabilities, but the bottleneck of 
building the internal symbolic representation by PRISM remains and blocks the successful scaling of the 
number of processes in the model. 

We have thus implemented an approach that instead generates the reachable state space of the 
symmetry-reduced model in an explicit manner and directly outputs the transition matrix and state labels 
of the DTMC in the format usable by MRMC. While our tool is tailored to this model, i.e., given the 
number of processes n and the parameters 70, J\ and v the tool enumerates the reachable state space and 
calculates the transition probabilities between the states, the matrix could likewise be generated by an 
explicit state probabilistic model checker like LiQuor [5l or even by an explicit state space generator for 
the PRISM input language. 

Improved scalability. Table |2] demonstrates the vast improvements in scalability resulting from our 
approach of directly generating the matrix and using MRMC for the calculation of the steady-state prob- 
abilities. The table shows the time spent by our tool to generate the matrix and the number of reachable 
states which were enumerated by our tool. This number corresponds to the number of rows and columns 
of the generated matrix. The second-to-last column lists the time spent by MRMC for calculating the 
steady-state probabilities for each state and writing the results to a file. The last column presents the time 
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Reachable states 


Time for 


Time for MRMC 


Time for 


Processes 


(rows/cols in matrix) 


matrix generation 


steady state calc. 


properties 


3 


33,768 


0.29 s 


0.13s 


0.52s 


4 


694,907 


6.90s 


2.68 s 


10.26 s 


5 


8,606,544 


90.07 s 


33.98s 


128.08s 


6 


58,911,750 


677.44s 


390.91s 


913.33s 


7 


213,497,440 


2,578.15s 


2,601.72s 


3,405.98 s 


8 


387,320,107 


4,605.24 s 


5,709.98 s 


6,568.41s 


9 


1,211,760 


11.42s 


4.20 s 


3.50s 


10 


189,311 


1.77 s 


0.68 s 


0.49 s 


11 


192,910 


1.82s 


0.69 s 


0.48 s 


100 


205,637 


2.17s 


0.79 s 


0.53 s 


1,000 


334,337 


7.96s 


1.16s 


0.86s 


10,000 


1,621,337 


255.14s 


5.18s 


4.01s 



Table 2: Statistics for directly generating the matrix for the symmetry-reduced model and calculating the 
steady state probabilities for n processes and distributions /o(5) = 1, /i(6) = 1 and v(40) = v(50) = A 
using the model checker MRMC. 



spent in a post-processing step to calculate the steady-state probabilities for a number of state properties 
from the output of MRMC, as detailed below. We have chosen this separate post-processing step to allow 
for a more detailed analysis of the results and to facilitate a simple implementation of the conditional 
long run operators of Q in the future. As our purpose was an initial evaluation of the potential of sym- 
metry reduction to this and similar models, both the matrix generator and the post processing tool are not 
yet heavily optimised for speed and space efficiency. 

As can be readily seen in Table [2j the complexity of the model for these parameters increases for up 
to 8 processes, with the model becoming drastically simpler afterwards. To illustrate this phenomenon 
and to gain further insights in the behaviour of the spinlock model when the number of processes and the 
distributions are varied, we have calculated the steady-state probabilities of a number of state properties 
using our tool and MRMC. 

Figure H] shows the effect of increasing the number of processes in our model. The figure on the left 
depicts the steady-state probability that at a given moment exactly k out of the n processes are in their 
non-critical section for our example distribution. As can be seen, increasing the number of processes 
at first increases the number of processes expected to be in their non-critical section because processes 
requesting the lock have a high probability to immediately acquire the lock and thus quickly return to the 
non-critical section. At some point the number of processes reaches a threshold where the lock becomes 
saturated. At this point, the probability that there is already a process waiting for the lock when the lock 
is released reaches 1 . This limits the number of processes that can simultaneously be in their non-critical 
section to a fixed number, which is related to the ratio of the longest time a process may spend in the 
non-critical section and the shortest time a process needs to pass through its critical section. The figure 
on the right in Figure [4] details the effect of lock saturation for an increasing number of processes. For 
the chosen values for % and v, saturation is reached for 10 or more processes, as the probability that in 
the long run some process is waiting for the lock at any given moment reaches 1 . This entails that each 
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Figure 4: The left diagram shows the steady-state probabilities that at a given moment there are exactly 
k processes in their non-critical section. Darker grey indicates higher probability, as indicated in the 
legend. The right diagram shows the steady-state probabilities that at a given moment (1) process Pi is 
"spinning", i.e., waiting for the lock and (2) that at least one process is "spinning". 2 to 100 processes. 
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(a) The steady-state probabilities that at a given moment 
there are two (or more) processes with distance equal to k 
in their non-critical section, for 2 to 100 processes and with 
y (5) = 1, y! (6) = 1 and v(40) = v(50) = \. Darker grey 
represents a higher probability as indicated in the legend on 
the right. 




Process 1 spinning 
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(b) Steady-state probabilities that at a given moment (1) 
process P\ is "spinning", i.e., waiting for the lock and (2) 
that at least one process is "spinning". 2 to 100 processes, 
7o(5) = 1, 7i(6) = 1 and v(50) = v(60) = \. The grey 
bar indicates the gap, where the model exceeded the 190GB 
RAM limit. 



Figure 5 



individual process, here represented by Pi , spends a larger and larger amount of its time waiting for the 
lock. 

Figure [5(a)] provides a different view on the same phenomenon. There, we analyse the regularity of 
the pattern in which processes pass through their non-critical section. The figure shows the steady-state 
probabilities that at a given moment two (or more) processes in their non-critical section have a distance 
of k time units, with no other process in between. E.g., two processes in their non-critical section with 
timer values of 12 and 16 and no process in between have a distance of 4. As can be expected, if the 
number of processes is sufficiently low to ensure that the lock is not saturated, the distances between 
the non-critical processes take a variety of values as the probabilistic choice of the time spent in the 
non-critical section and the two different lengths of the critical sections "shuffle" the order and distance 
of the processes. Increasing the number of processes then leads to a much more regular structure, as 
the regularity with which processes pass through the lock induces less variety in the distances between 
processes. 
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Varying the distribution. We have seen that our approach of using symmetry reduction and the model 
checker MRMC allowed us to successfully scale the number of processes for the example distribution 
v(40) = v(50) = o- Unfortunately, for other, more challenging distributions considered in [2], the picture 
is not as rosy Figure [5(b)] shows some results for the distribution v(50) = v(60) = A. As can be 
seen, the values for n = 7 to 9 processes could not be calculated, as the corresponding matrix could not 
be generated before reaching the memory limit of 190 GB. Similar gaps where generating the matrix 
exceeds the available RAM exist also for other distributions we considered. 

Scalability of saturated locks. The drastic improvements in scalability beyond our expectations and 
the collapse in complexity that we observe is linked to the complete or almost complete saturation of 
the lock. Increasing the number of processes further only increases the number of processes that wait 
spinning for the lock. 

Of course, spinlocks are known to not scale well in high-contention scenarios and are therefore not 
widely used in situations where lock contention can become high. For practical purposes, parameters 
that lead to lock saturation then strongly indicate that locking realised via a spinlock is probably not the 
right choice and more sophisticated locking schemes or even architectural redesign of a given system 
is required. However, we expect that the approach presented here should apply equally well to other 
synchronisation primitives for highly contended resources and other scenarios such as the analysis of 
thread pools where by construction the majority of worker threads await further requests. 



4 Conclusions and Future Work 

In this paper, we have investigated how symmetry reduction helps scaling a previous case study on model 
checking conditional long run properties of low-level operating-system code. Using a model specific 
adaption of the technique of generic representatives, we were able to significantly scale the number of 
processes in the model beyond the point were adding more processes does not significantly increase 
the complexity of the analysis anymore. This allows an analysis of models with 10,000 processes and 
beyond using the model checker MRMC. There are still situations where the analysis proves elusive for 
certain number of processes due to the high number of reachable states. 

Future work. The results presented in this paper represent work in progress and allowed us to evaluate 
the potential benefit of using symmetry reduction techniques for the scalability in the number of processes 
of a probabilistic analysis of locking mechanisms and similar low-level constructs. We plan to further 
improve the time and space efficiency of the tool we use to generate the state space. We also plan to 
investigate refined notions of symmetry reduction, the compatibility with symbolic methods, i.e., the 
encoding of the states and corresponding variable orders, as well as other reduction techniques such as 
bisimulation quotienting/lumping lfl4l ITSTl . As our results show the feasibility of an analysis approach 
using symmetry reduction and the model checker MRMC, we further plan to extend MRMC by the 
capability to calculate conditional long-run properties of the type considered in Q. 

A second topic for future work are more complicated case studies. Interesting candidates are thread 
pools and queue-based locks (U [20J as they enforce a strong dependency between waiting processes. 
Particularly interesting due to their potential feedback on the design and use of these algorithms are 
reactive locking schemes lfl9l . Probabilistic model checking results could guide when the lock switches 
to an alternative implementation. Overcoming the complexity gap remains a top priority. 
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